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1. INTRODUCTION 


1.1 Objectives 


Both the military and civilian segments of aviation axe placing an increasing reliance on flight simulators for 
aircraft research and development and for pilot training and proficiency maintenance. Because of the increasing 
sophistication and associated costs of available simulation devices, issues relating to trade-offs between simulation 
fidelity and costs have risen to highly visible levels. When determining the simulator configuration to meet a 
specific goals, the designer must consider the need for particular cueing devices as well as the requisite level of 
fidelity, and he should understand the constraints placed upon man-machine system performance (including 
performance/workload trade-offs), as well as costs, imposed by that level of simulation fidelity. 

To address these issues, NASA-LaRC undertook a combined experimental and analytical program to develop 
data and methodologies for the evaluation and specification of simulator validity and fidelity. The experimental 
program involved ground-based simulator experiments at NASA-LaRC facilities and in-flight tests, using an "F-14" 
aircraft, performed at Edwards Air Force Base. BBN’s role in this effort was to first analyze the simulator results 
with the aid of the optimal control model (OCM) for pilot- vehicle systems for the purposes of (1) "calibrating" the 
model as required for the subject pool and task environment, and (2) providing an analytic explanation for 
performance differences, if any, among the cuing conditions explored in the simulation. This analysis was to be 
followed by an analysis of the in-flight tests, again with the goal of using the model to provide an analytic 
framework for whatever performance differences might be found between ground-based and in-flight performance. 

A substantial portion of BBN’s effort was devoted to obtaining an accurate and workable linear model of the 
simulated air-to-air combat task. The plan of attack described above was not completed because of difficulties 
encountered in the production of the NASA data base, particularly with respect to the in-flight tests. During the 
contract performance period, however, the Calspan Corporation provided BBN with a data base obtained from a 
combined ground-based and in-flight simulation study of simulator fidelity using an NT-33A variable-stability 
aircraft. 1 Given this situation, the best available course of action in support of the goals of the contract was to 
perform model analysis of the NT-33A data base. 


'Calspan Corporation, Air Force Contract No. F33615-83-C-3606 


In effect, BBN’s effort under the NASA contract was conducted in two phases: (1) model development and 
data analysis relevant to the NASA simulation study, and (2) analysis of the NT-33A data base. The Phase 1 effort 
has been described in an interim report (Levison and Baron, 1984) and in subsequent progress reports. The results 
of this phase are summarized briefly below; the bulk of this report is devoted to a summary of the Phase 2 effort. 


1.2 Review of the Phase 1 Effort 

The task explored in the NASA simulation and in-flight tests involved an "F-14" aircraft tracking a target 
aircraft executing a 3g wind-up turn. Once steady-state undisturbed tracking was achieved, the vertical 
displacement of the reticle (sight) of the pursuing aircraft was disturbed by a random appearing signal. The 
simulator experiments duplicated this task under a variety of cuing conditions. Data were obtained using two 
ground-based simulators: the moving-base Visual Motion Simulator (VMS) and the fixed-base Differential 
Maneuvering Simulator (DMS). Four experimental conditions were explored in each simulator: with and without 
"motion", and with and without g-seat cuing. The "motion" condition represented whole-body motion cuing in the 
VMS and helmet-loading in the DMS. The same subjects performed in both simulators as well as in the "F- 14". 

As noted above, much of the Phase 1 BBN effort was devoted to formulating a model of the air-to-air tracking 
task, including (1) modeling the target kinematics, (2) developing the linearized line-of- sight equations, (3) 
reducing the dimensionality of the overall model, (4) validating the math model against the NASA simulation, and 
(5) performing a sensitivity analysis with the model to explore the effects of assumptions concerning the simulator 
and/or the pilot. These results have been summarized by Levison and Baron (1984). 

Statistical analysis of the tracking scores obtained in the NASA simulation studies revealed the following 
general trends. 

• Effects of non-visual cuing ("motion" or "g-seat") were neither large nor statistically significant. 

• Differences across subjects were relatively large and statistically significant. 

• Although not found to be statistically significant, the VMS/DMS differences in lateral-axis LOS errors 

were relatively large. 

The lack of differences across cuing conditions is consistent with post-experiment model analysis which 
showed a predicted lack of sensitivity of line-of-sight tracking error to the presence of motion cues (Baron and 
Levison, 1984). 
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1.3 The NT-33 A Study 


The major objective of the NT-33A study was to explore the effects of control system delays on closed-loop 
performance in ground-based and in-flight simulations of an attitude tracking task. Pre-experiment model analysis 
was performed by BBN to help specify certain experimental variables: primarily, parameters of the external forcing 
functions to be used in the steady-state tracking segments of the NT-33A simulations. The pre-experiment analysis 

has been documented by Levison and Papazianf 1986): major predictions yielded by this analysis were: 

1. modest increases in tracking error score with increasing delay, 

2. larger delay effects for the generic high-performance fighter flown aggressively ("F-16") than for the 

generic heavy transport ("C-141") flown in a more relaxed manner, 

3. relatively small differences (less than 10%) between performance in-flight and in the ground-based 
simulations, and 

4. similar relative effects of experimental variable on performance in the pitch and roll axes. 

As shown in this report, these pre-experiment predictions were largely confirmed by the manned simulation 
experiments. 

Post-experiment analysis included statistical analysis of steady-state tracking performance metrics and model 
analysis using the same optimal control model (OCM) employed earlier in the study. The reader is assumed to be 
familiar with the structure and application of this model. The major objectives of the model analysis were to help 
explain any differences that might arise between experimental results and the results predicted by pre-experiment 
analysis, and to explore the utility of the OCM as a potential tool for interpolating the results of simulation 
experiments designed to explore time delay effects. 

A subset of the NT-33A experimental conditions was analyzed by BBN. Pitch- and roll-axis performance in 
single-axis target-tracking tasks was analyzed for the following combinations of experimental variables: (a) "F-16" 
and "C-141" aircraft dynamics, (b) zero and 180 msec simulation delay added to the irreducible simulation delay of 
about 100 msec, and (c) ground-based and in-flight simulations. In addition, the combined-axis performance in roll 
and pitch were analyzed for the following combination of experimental variables: (a) "F-16” and "C-141" dynamics, 
(b) zero and 180 msec added delay, and (c) target and disturbance inputs. In-flight performance for combined-axis 
tasks was not analyzed by BBN because data were not available for all test pilots. 
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"Single-axis" tasks were tasks in which an external forcing function was applied to either the roll or the pitch 
axis, but not both. Forcing functions were applied to both axes in the "combined-axis" tasks. In all cases, the pilot 
was required to control both the pitch and roll axes. 


1.4 Organization of this Report 


The remainder of this report presents the results of the analysis performed by BBN on the NT-33A data. 
Procedures used in performing both statistical analysis and model analysis on the experimental data are reviewed in 
Chapter 2. Major results are presented in Chapter 3, and the discussion provided in Chapter summarizes these 
results, outlines a procedure for a combined analytical and experimental approach to studying simulator fidelity 
issues, and suggests areas of further study aimed toward development and validation of this approach. 
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2. ANALYSIS PROCEDURES 


2.1 Statistical Analysis of Tracking Performance 


Means and standard errors were obtained for closed-loop performance metrics and for pilot frequency 
response measures as described below. 

2.1.1 Closed-Loop Performance Metrics 

Primary data reduction was performed by Calspan; BBN performed the subsequent averaging across 
replications and across subjects. Within-trial mean and standard-deviation (SD) scores were first computed for all 
important problem variables. These scores were then averaged across the two replications/subject to provide 
average performance measures for each subject for each of the experimental conditions considered in the BBN 
analysis. Finally, population means and estimated standard errors of the population means were computed for each 
condition of interest. The summary statistics reported later in this report are the means and standard errors of the SD 
scores for tracking error, control force, and control force-rate. 

2.1.2 Frequency-Response Metrics 

Pilot frequency-response metrics consisted of the describing function — shown as "gain" (more properly, 
"amplitude-ratio") and phase-shift" curves, and curves of the pilot’s "remnant" (the spectral density of that portion of 
the pilot’s control or "stick" force that is not linearly correlated with the external forcing function). 

It was originally intended that analysis techniques appropriate to sum-of-sines (SOS) inputs would be 
employed (Zacharias and Levison, 1979). Because the nominal SOS input used in this study had significant power 
at off-nominal input frequencies (i.e. "sideband power"), the analysis technique was modified to be appropriate to 
inputs continuous in frequency. Specifically, the pilot’s describing function was estimated at each nominal SOS 
frequency as: 
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where H is the describing function expressed as a complex number, and I,E, and U are the Fourier coefficients of the 
external input, the tracking error, and pilot’s control response, respectively, obtained from fast-Fourier analysis. The 
overstrike signifies averaging of the cross-power terms over a frequency range of approximately 1/4 octave centered 
about the nominal SOS frequency. 

The remnant power was computed in the vicinity of each SOS input frequency as: 


|ui *| 2 

Remnant = |U | 2 - 

ITT 2 


Because of the significant amount of pilot remnant, intra- and inter-subject averaging of the describing 
functions was performed using techniques recently developed to maximize the reliability of the averaging process 
(Levison, 1985). Individual describing function as defined above were not averaged: instead, averaging was 
performed on the cross-power quantities, first across replications and then across subjects. The inter-subject average 
describing function was then computed as shown in Equation 1, where the numerator and denominator terms now 
represent inter-subject averages. The resulting complex number H was the converted to equivalent gain and phase 
shift. 


Because phase shift is defined only within a cycle, the OCM was used to generate a theoretical reference 
curve to "unwrap" the phase curve. This unwrapping procedure has been described by Levison, Junker, and Kenner 
(1985). 

Estimated standard errors of the mean gain and phase shift were computed from the variability of the 
cross-power spectral measures as described by Levison (1985). Inter-subject statistics on the remnant estimates 
were obtained via the same procedures adopted for the closed-loop performance metrics. 
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2.2 Model Analysis 


A major objective of the model analysis was to explore the potential of the OCM as a tool for interpolating the 
results of manned simulations, rather than to obtain the best curve fit possible. Therefore, independent "pilot- 
related" model parameters were not subjected to the type of unrestricted gradient search identification procedure that 
has been used appropriately in the past for quantifying operator capabilities (Levison, 1981). Instead, model 
analysis was initiated with parameters selected from previous laboratory studies, and subsequent parameter changes 
were made only as required to provide an acceptable subjective match to the data for certain baseline conditions. 
Predictions were then obtained for other experimental conditions with no further changes in pilot-related parameters. 


In order to maximize computational efficiency, most of the model analysis was conducted to provide 
comparisons with the average performance of the three test pilots. There were substantial inter-subject differences 
however, and some analysis was conducted to explore these differences. 

In this section of the memo we describe the procedure by which independent model parameters were selected. 
Interpretation of the specific values selected is provided in Chapter 4. 


Non-zero values were assigned to the following pilot-related model parameters: 

• Motor Time Constant. This parameter reflects bandwidth limitations imposed by the pilot/stick 
interface. Values on the order of 0.09 to 0.13 seconds are typical for force sticks and optimal control 
gains. Larger values have been found for sticks having significant displacement characteristics 
(Repperger and Levison, 1984). 

• Operator Delay. A pure transport delay is associated with the operator’s response characteristics. This 
OCM independent parameter is generally not influenced by the tracking task; a value of 0.2 seconds is 
typical. When the tracking task contains display-related delays that show up in the computation of the 
pilot describing function, such delays should be included in the "operator delay" to allow a meaningful 
comparison of measured and predicted operator phase shift. For this study, operator delay was fixed at 
0.22 seconds — 0.20 for the operator and 0.02 for display delay. 

• Penalty on Control Force. For tasks in which the control gain is optimal, there is generally no need to 
assign a performance penalty (i.e., a non- zero "cost coefficient" in the quadratic performance index) to 
control force or displacement Where significant forces are required, however, a specific penalty 
associated with control force (or possibly control rate) improves the match to operator performance 
(Levison, 1984). The control penalty is expressed here as the numerical value of the control SD score 
that contributes one unit of cost to the performance index. (Note: one unit of cost is assigned to an SD 
score of 1 degree for roll or pitch tracking error.) 

• Residual Noise on Error Perception. The OCM allows the user to specify a "residual noise" standard 




7 



deviation for each of the perceptual variables used by the pilot to reflect the effects of threshold-like 
phenomena such as perceptual resolution limitations and "indifference thresholds". Residual noise is 
usually set to zero for laboratory tracking tasks that use symbolic displays with optimal display gains. 
In general, however, nonzero residual noise should be specified for non-ideal perceptual environments 
such as real-world visual scene cues (Levison and Warren, 1984), or when the operator is indifferent to 
errors below a certain level. 

• Observation Noise Ratio. Pilot remnant is modeled largely as an observation (perceptual) noise 
process. Except for the residual noise term, observation noise is assumed to scale with the rms value of 
the perceived variable. An "observation noise ratio" of around -20 dB is typical for laboratory 
experiments. Based on the results of a recent laboratory study of time-delay effects (Levison and 
Huggins, 1986), this parameter was fixed at -19 dB for this analysis. 

• Motor Noise Ratio. Except for tasks using very large control sensitivities, or human operators with 
substantial neurological "tremor", this term is negligible. It was fixed at -90 dB for this analysis. 

• Internal Motor Noise Ratio. To reflect certain limitations on the operator’s ability to predict the effects 
of his control input on vehicle response, we assign a small but non-negligible value to the "internal" 
motor noise (i.e.. the pilot’s perception of his motor noise variance). This parameter was fixed at -44 
dB based on the Levison and Huggins study. 


Three pilot-related parameters were varied during the course of this analysis: motor time constant, the control 
penalty, and residual noise on tracking error. The remaining parameters were kept fixed as described above. 


For the purposes of model analysis, the "baseline" condition was defined as single-variable tracking, target 
input, ground-based simulation, zero added delay. There were thus four experimental baseline tracking tasks: pitch 
and roll, "F-16" and "C-141". The approach to model analysis was to first obtain two "calibration" sets of 
independent model parameters — one providing the best joint match to "F-16" and "C-141" performance in roll, and 
one providing a joint match to pitch-axis performance for the baseline conditions. With parameters held fixed at the 
appropriate baseline values, predictions were then obtained for (1) the effects of additional delay on performance, 
and (2) performance differences between ground and in-flight simulation. 


Values used in the pre-experiment analysis - shown in the first column of Table 2-1 — were used in an initial 
attempt to match the roll-axis data. These values proved to be too optimistic in terms of predicting average 
performance. In particular, the experimental tracking errors scores were substantially larger than predicted, the 


control scores were lower than predicted, and the remnant was greater than predicted. 


Table 2-1: Independent Parameters Adjusted During the Model analysis 


Parameter 


Initial Roll 


Motor Time Constant, sec 
Control for Unit Cost, pounds 
Rms Residual Noise, degrees 


0.12 0.20 
0 . 0 . 

0. 75. 


Pitch 

0.15 

20 . 

2.25 


The predicted control and control-rate scores were brought into general agreement with the data by increasing 
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the motor time constant to 0.20 seconds. This action had the additional effect of increasing the predicted tracking 
error score, but not by the amount needed to match experimental results. 

Examination of the performance of individual test pilots showed substantial inter-subject differences. Pilot A 
was the best performer in terms of minimizing the roll-erTor SD score; his performance could be matched with 
values near those initially selected. As noted above, considerations of efficiency dictated that the bulk of the 
analysis be performed on average performance, and the discussion here is relevant to average performance. Later in 
this report we document some of the inter-subject differences. 

One or more model parameters related to pilot remnant had to be modified to provide better matches to 
experimental tracking error scores and remnant spectra. Increasing the observation noise tended to provide a better 
match to the remnant at mid and high frequencies, whereas including the residual noise improved the match at low 
frequencies. Both methods were initially explored, but the residual-noise adjustment seemed to provide slightly 
better predictive capabilities and is therefore the method that was pursued. 

A residual noise of around 75 degrees was required to provide an acceptable match to the baseline data (i.e., 
predicted SD scores within 10-15% of the experimental scores) for the roll axis. This value was much larger than 
would be needed to accommodate an indifference threshold of 5 degrees that one might infer from the instructions to 
the test pilots. As we show later in this report, this large value might reflect, in part, significant nonlinearities in the 
pilot’s response behavior. 

The pitch-axis baseline tasks were then modeled. The motor time constant of 0.2 seconds obtained in the 
roll-axis match was reduced to 0. 15 to provide a better match to the control and control-rate scores, and the residual 
noise was readjusted to match tracking error. Introduction of the control penalty for the pitch-axis task provided a 
somewhat better match to the "F-16"/"C-141" performance differences. (A subsequent re-analysis of the roll-axis 
task showed that the same control penalty had negligible effect on predicted roll-axis performance.) 

Once calibrated for the baseline tasks, the pilot-related model parameters considered above were held fixed as 
the effects of experimental variables were predicted. Changes in experimental conditions were modeled as follows; 

• Vehicle delay. The addition of delay to the simulated vehicle’s control-system delay was modeled by 
simply incrementing the "vehicle delay" parameter of the OCM by the same amount. (A baseline 
vehicle delay of 0.08 seconds was included to reflea irreducible simulator delays exclusive of display- 
related delay.) 

* Ground/flight differences. The parameterization defined above pertains to the ground-based simulation. 

To account for the effects of in-flight simulation, display variables associated with perception of wbole- 
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body motion cues were assigned fractional attentions of unity. These variables were vehicle roll rate 
and roll acceleration for roll-axis tracking, and vehicle pitch rate, pitch acceleration, and normal 
acceleration for pitch-axis tracking. 

Combined-axis tracking. Implicit in the parameterization defined above is the assumption that the 
pilots could devote nearly full attention to the control axis containing the external forcing- function for 
the single-axis tasks. For combined-axis tracking, we assume that the pilot’s must allocate attention in 
roughly equal proportions to the pitch- and roll-axis tracking tasks. Combined-axis tracking was 
therefore modeled by increasing the observation noise ratio by 3 dB (equivalent to assigning fractional 
attentions of 0.5 to all perceived variables). 
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3. Principal Results 


The results of the statistical and model analysis are presented together in this chapter. Results are organized 
as follows: (1) closed-loop performance metrics, (2) frequency-response measures, and (3) nonlinear analysis of 
selected time histories. 


3.1 Closed-Loop Performance Measures 


Means and standard errors for tracking error, stick force, and stick force rate SD scores are tabulated in the 
appendix for the conditions analyzed by BBN. Also tabulated are the corresponding model predictions. 

Effects of added delay and simulator mode on measured and predicted SD scores are shown in Figure 3-1 for 
the roll-axis task and in Figure 3-2 for the pitch-axis task. These results pertain to single-axis target-following tasks. 

The trends of the tracking error scores were generally as predicted by the pre-experiment model analysis (and 
confirmed by the post-experiment model analysis shown in the figures): 

• Addition of 180 msec delay produced a modest increase in the SD score. 

• Delay effects are relatively greater for the "F-16" dynamics than for the "C-Ml". 

• Ground-base simulation produced somewhat larger error scores than in- flight simulation, but the 
effects of simulator mode on performance were less than the effects of control-system delay for the 
delays considered in this analysis. 

As in a previous study of time-delay effects (Levison and Huggins, 1986), the model tended to slightly 
underestimate the relative increase in error due to added delay. Model/data discrepancies of this sort were, however, 
within the experimental standard error and cannot be considered statistically significant. 

The largest model-data discrepancy concerning tracking error scores - the only difference likely to be 
statistically significant - occurred for "F-16" pitch-axis tracking (Figure 3-2a), where the model underestimated 
differences between in-flight and ground simulator performance. This trend has been seen in some previous 
experiments in which the benefits of whole-body motion cuing have not been fully accounted for by the kind of 
simple informational treatment conducted here (Levison, Lancraft, and Junker, 1979). Ground/flight differences for 
the remaining error scores ("C-141" pitch, "F-16" and "C-141" roll), however, are predicted rather closely. 
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a) "F-16", ROLL ERROR 


b) "C-141", ROLL ERROR 



c) "F-16", STICK FORCE 


d) "C-141", STICK FORCE 


_ 4 - 4 

V) 

TJ 

C 



Mode G F G F Mode G F G F 

Delay 0 180 Delay 0 180 


Figure 3-1: Effects of Delay and Simulator Mode on SD Performance Scores, Roll Axis 
Target-tracking task. 

Average of 3 subjects, 2 replications/subject. 

G = ground-base, F = in-flight. 
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a) "F-16", PITCH ERROR b) "C-141", PITCH ERROR 



Delay 0 180 Delay 0 180 

c) "F-16”, STICK FORCE d) "C-141", STICK FORCE 



Figure 3-2: Effects of Delay and Simulator Mode on SD Performance Scores, Pitch Axis 

Target -tracking task. 

Average of 3 subjects, 2 replications /subject . 

G = ground-base, r = in-flight. 
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Figure 3-lc and 3-2c show little measured or predicted effects of experimental variables on the stick SD score 
for "F-16" tasks. Effects of both delay and simulator mode are apparent in the "C-141" experimental data (Figures 
3- Id and 3-2d). The model mimicked the noticeable mode difference for the 'C-141", zero-delay roll-axis task, but 
it did not predict the observed effects of delay on the "C-141" stick score. 

Table 3-1 provides a quantitative summary of the relative effects of the experimental factors of control-system 
delay and simulator mode. Entries under the "delay" heading were obtained by taking the ratio of the population- 
average error SD score for the 180-msec condition to the average error score for the zero-delay condition. Each 
entry in the table is the average of four such ratios (pitch and roll, ground and flight). Similarly, entries under the 
"simulator mode" heading were obtained by averaging the rations of the "ground" error scores to the "flight" error 
scores, averaged over the two axes and the two delay conditions. All data in this table pertain to the single-axis, 
target-tracking tasks. 

Table 3-1: Effects of Experimental Factors on Average Performance Ratios 

Delay Simulator Mode 

"F-16" "C-141" "F-16" "C-141" 

Data 1.25 1.21 1.18 1.03 

Model 1.19 1.15 1.06 1.06 

Averaged over single-axis, target-tracking tasks. 

Three subjects, two replications/subject. 

Table 3-1 confirms the qualitative impressions obtained from Figure 3-1 and 3-2. In terms of the proportional 
increase in tracking error scores, the addition of 180 msec delay had a greater effect than the transition from flight to 
ground, and the effects of delay were slighdy greater for the "F-16” than for the "C-141". The model replicated 
these trends, although it did not predict the full extent of the delay effects. 

Comparisons of experimental single- and combined-task performance for the zero and 180-msec delay 
conditions are provided by Figures 3-3 and 3-4 for roll and pitch performance, respectively. Also shown are model 
predictions for the 2-axis, zero-delay conditions. 

Combined-task performance scores for the roll-axis were nearly identical to the single-task scores. Inspection 
of the standard errors provided in Table A-l of the Appendix suggests that the small differences shown in Figure 3-3 
were not statistically significant. 

Taskloading had a more pronounced effect on pitch-axis performance. Figure 3-4 shows that combined-axis 
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a) F-16, Roll Error 


b) C-141, Roll Error 
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0 180 

Delay (msec) 


c) F-16, Stick Force d) C-141, Stick Force 




Figure 3-3: Effects of Delay on Single- and Combined-Task SD Performance Scores, Roll Axis 
Target-tracking task. 

Average of 3 subjects, 2 replications/subject. 
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stick scores were 25% to 35% greater than single-axis scores for both dynamics and delay conditions. Effects of 
taskloading on pitch error was less consistent. The error scores for the zero-delay conditions were the same for the 
1- and 2-axis tasks, but the addition of 180 msec delay caused larger performance degradations for the combined- 
axis tasks. 

The horizontal bars in Figure 3-3 and 3-4 show that the model overestimated the differences between 
combined-task and single-task performance, predicting differences in the error SD score on the order of 20% to 30% 
when, in fact, experimental differences were negligible. Because of the clear discrepancy between model and data 
with regard to taskloading effects, model analysis was not performed for the 180-msec delay condition. 

The tendency of the model to predict taskloading effects on tracking error when none were found 
experimentally does not signify a structural defect of the OCM, but it does suggest a slight revision in the procedure 
for selecting pilot parameters. Discussion of this issue is deferred to Chapter 4. 

Effects of system delay on combined-task disturbance-regulation scores are shown in Figures 3-5 and 3-6 for 
roll and pitch, respectively. Although contract resources did not allow modeling of the disturbance tasks, 
comparisons of experimental results can be made to the pre-experiment model predictions documented by Levison 
and Huggins (1986). 

As predicted by the model, the effect of delay on-"C-141" tracking error was about the same for the 
disturbance-regulation task as for the target-following task. The trend of the "F-16" results was the reverse of that 
predicted, however. Pre-experiment model analysis indicted that delay would have a substantially smaller effect on 
pitch-axis performance than on roll-axis, whereas the experimental data showed the opposite trend. Two factors may 
account for this model/data discrepancy: (1) the pre-experiment model analysis was not conducted with independent 
parameters that provided the best match to the baseline data, and (2) the forcing functions applied experimentally 
differed in certain respect from those assumed for the initial model analysis. Additional model analysis of the 
disturbance-tracking results would be needed to resolve this apparent discrepancy for the "F-16" task. 

Discussion of the model has concentrated largely on the effects of delay and taskloading. It should also be 
noted that the OCM accounted for performance differences between the two simulated aircraft that were considered 
in the BBN analysis. 
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Figure 3-5: Effects of Delay on SD Performance Scores for Disturbance Regulation, Roll Axis 
Average of 3 subjects, 2 replications/subject 
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a) F-16, Pitch Error b) C-141, Pitch Error 




c) F-16, Stick Force d) C-141, Stick Force 




Figure 3-6: Effects of Delay on SD Performance Scores for Distuibance Regulation, Pitch Axis 
Average of 3 subjects, 2 replications/subject. 
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3.2 Frequency-Response Measures 


Figures 3-7 through 3-12 show the effects of three experimental variables (dynamics, delay, and simulator 
mode) on pilot frequency response. Discrete symbols and smooth curves indicate experimental data and model 
results, respectively. A gain of 0 dB represents one pound control force per degree display error, and a remnant 
level of 0 dB represents a control power density level of one pound 2 per radian/second. All results pertain to 
single-axis, target-tracking tasks. 

All frequency-response measurements are shown, regardless of the the standard error. In general, all 
frequency measurements below 8 rad/sec are "reliable" in terms of a low standard error. The reliability of 
measurements at higher frequencies fluctuated quite a bit from one condition to the next. In general, measurements 
at the highest SOS frequency should be regarded with caution. 

Figure 3-7 shows that plant dynamics influenced primarily the pilot’s response gain, the phase shift at mid 
frequencies, and the remnant spectrum in the roll axis. The gain and remnant differences reflect, for the most part, 
the different control gains provided for the two simulated aircraft. The pitch-axis data provided in Figure 3-8 show 
different frequency dependencies for the two vehicles as well as differences in overall gain and remnant levels. In 
particular, the "F-16" dynamics showed a more pronounced high-frequency peak in the gain response and a flatter 
phase response up to about 10 rad/sec than the "C-141" dynamics. Differences in pilot response across simulated 
vehicle and across axes of control were mimicked by the model. 

As might be expected from the preceding discussion of closed-loop performance metrics, the effects of delay 
and simulation mode on pilot frequency response were substantially less than the effects of simulated aircraft 
dynamics. Figures 3-9 and 3-10 show that the qualitative effects of adding delay, in either axis, were to (1) slightly 
decrease pilot gain at low and mid frequencies, (2) decrease the frequency at which the gain curve peaks, (3) 
decrease phase lag (or increase phase lead) at mid frequencies, and (4) increase pilot remnant. As correctly 
predicted by the model, the effects of delay on gain and phase shift were more pronounced for the pitch axis 
response. 

Figures 3-11 and 3-12 show some reduction in remnant and phase lag with the transition from ground to 
flight. The remnant curve may be interpreted as indicating a slightly wider response bandwidth for the in-flight 
tasks. The only sizeable effect predicted by the model (and not revealed in the data) was a mode-related difference 
in phase shift at the lowest measurement frequencies. 
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Figure 3-7: Effects of Dynamics on Pilot Frequency Response, Roll Axis 

Average of 3 subjects, 2 replications/subject. 

Single-axis, zero-delay, ground-based, target-tracking task. 
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Figure 3-8: Effects of Dynamics on Pilot Frequency Response, Pitch Axis 

Average of 3 subjects, 2 replications/sub ject . 

Single-axis, zero-delay, ground-based, target-tracking task. 
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Figure 3-9: Effects of Delay on Pilot Frequency Response, Roll Axis 

Avarage of 3 subjects, 2 replications/subject. 
Single-axis, "r-16", ground-based, target-tracking task. 
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Figure 3-10: Effects of Delay on Pilot Frequency Response, Pitch Axis 

Average of 3 subjects, 2 replications/sub ject . 
Single-axis, ’T-16", ground-based, target-tracking task. 
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Figure 3-11: Effects of Simulator Mode on Pilot Frequency Response. Roll Axis 

Average of 3 subjects, 2 replications/subject. 
Single-axis, "F-16", zero-delay, target -tracking task. 
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Figure 3-12: Effects of Simulator Mode on Pilot Frequency Response, Pitch Axis 

Average of 3 subject*, 2 replications/subject. 
Single-axis, zero-delay, target-tracking task. 
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3.3 Nonlinear Analysis 


The exploration of potential nonlinearities in the operator’s response behavior was motivated by the unusually 
large values for the "motor time constant" and remnant-related parameters needed to provide an acceptable match to 
the average performance of the three test pilots. Examination of the individual performance of the three test pilots 
showed considerable differences between the best-performing pilot (in terms of minimizing rms roll error) and the 
other two pilots. These results prompted the suspicion that the response strategies of the two less-well-performing 
pilots might have contained significant nonlineanties, and that these nonlinearities might have resulted in the 
apparent reduced bandwidth and larger tracking error characteristic of these pilots. 

Nonlinear analysis consisted of visual inspection of time histories, and statistical analysis of the probability 
densities, of one trial each for each of the three test pilots. The experimental condition represented by these trials 
was roll-axis tracking, ground-based simulation, "F-16" dynamics without additional time delay, with an external 
sum-of-sines target-command signal active in the roll axis only. 


3.3.1 Visual Inspection of Time Histories 

Initial visual inspection of the pilot’s output response ("stick") revealed substantial noise at the sampling 
frequency (50 Hz), which was felt to be due largely to the quantization of the stick amplitude imposed by the 
recording technique. In order to prevent this noise from masking important response trends, the stick signal was 
digitally filtered with a rectangular time window of 0.08 seconds. That is, each data point was replaced by the 
locally-averaged stick amplitude. This filtering operation significantly reduced high-frequency noise without 
apparent distortion to signal components within the bandwidth of man/machine response. 

Time histories generated by the best-performing (Pilot A) and worst-performing (Pilot B) were inspected. 
Numerical printouts of the stick time histories were initially scanned for the appearances of intervals of little or no 
stick motion ("flat spots"). Five-second intervals containing these occurrences were then plotted for inspection. 

In order to provide references against which to compare the experimental time histories, the following model 
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analysis was performed. First, the various performance metrics obtained from the two experimental trials were 
matched with the steady-state optimal control model (OCM). The simulation implementation of the OCM 
(Kleinman, Baron, and Berliner, 1977) was then used to generate stick time histories for the 5-second time segments 
of interest, using the appropriate Calspan experimental forcing function as the driving function. If the model were to 
match the pilot’s linear characteristics perfectly, and if the effects of pilot remnant remnant were negligible, the 
model and experimental stick time histories would be identical. Given the existence of remnant and the difficulty of 
a perfect match, the model-generated time histories are intended more as a qualitative reference. Specifically, one 
can determine whether or not the model predicts the response trends exhibited by the data. 


Table 3-2 shows the independent "pilot-related" model parameters that were found to provide an acceptable 
match to the data Parameters shown for pilot A were initially selected on the basis of a recent BBN laboratory 
study on time-dfelay effects (Levison and Huggins, 1986). Only the motor time constant needed adjustment to 
provide a qualitatively acceptable match to the performance of Pilot A. To match the data of Pilot B, however, 
significant changes were made to remnant-related parameters, and the motor time constant was further increased to 
reflect the lower response bandwidth of this subject. The observation noise/signal ratio was increased by 4 dB, and a 
residual noise of 15 degrees — representative of a visual or "indifference" threshold of 5 degrees - was included. 
This residual noise value was consistent with the instructions given to the pilots, from which the pilots could infer 
that it was not necessary to maintain roll errors below 5 degrees. Residual noise was not needed to match the 
performance of Pilot A, however. 

Table 3-2: Independent Model Parameters 


Parameter 


Pilot A Pilot B 


Motor Time Constant, sec 
Operator Delay, sec 
Residual Noise, Roll Angle, deg 
Observation Noise/Signal Ratio, dB 
Internal Motor N/S Ratio, dB 


0.16 0.30 

0.200 0.225 

0.0 15.0 

- 19.0 - 15.0 

- 44.0 - 44.0 


Time histories for Pilot A are shown in Figure 3-13. Four plots are shown: (a) the experimental stick time 
history , (b) the stick time history predicted by the model, (c) the experimental error time history, and (d) the 
predicted error time history. Because the model analysis and experimental simulation used different sign 
conventions, the predicted error waveform shown in Figure 3- 1 3d has been inverted to facilitate comparison with the 
other curves shown in Figure 3-13. 
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Comparison of Figures 3-13a and 3-13b shows that the predicted stick response is qualitatively (and to some 
extent, quantitatively) similar to the experimental stick response. As noted above, these waveforms cannot be 
expected to overlap exactly because of the stochastic response components (remnant) in both the experimental and 
predicted time histories. Nevertheless, one expects to see some correlation between the two because the same 
external forcing-function waveform has driven both the experimental and modeled systems. 

The experimental stick response appears to show a tendency to remain relatively flat for an interval of about 
0.5 seconds beginning around 31.5 seconds into the run. The model prediction does not support this tendency, 
suggesting evidence of a response nonlinearity. Note, however, that the experimental error time history (Figure 
3-13c) also shows a relative flat spot over the same interval. If the pilot’s response strategy were to act largely as a 
gain on roll error, this apparent pause in the pilot’s response could be interpreted as a linear response to the error 
signal. 

Stronger evidence of nonlinear response behavior is revealed by the time histories of Pilot B in Figure 3-14. 
Figure 3- 14a shows a relative pause in response activity of about 1 second, beginning around 43.5 seconds into the 
run. The error signal plotted below does not reveal a similar flat spot, but rather a local maximum as the error 
reverses direction and heads toward zero. Furthermore, the model shows a more peaked response pattern (Figure 
3-13b) in response to the same type of error pattern (Figure 3-13d) as found in the data. In this case, the pilot's 
response is less indicative of a linear response strategy and more suggestive of a "move-and-wait" strategy in which 
the pilot maintains a relatively constant control force that is sufficient to cause the error to begin to decrease. Other 
segments of this pilot’s response time history (not shown here) revealed qualitatively similar patterns. 

3.3.2 Statistical Analysis of Stick-Rate Amplitude 

Objective analysis of selected time histories was performed to explore quantitative manifestations of the 
move-and-wait episodes subjectively found from visual inspection. Specifically, we tested the hypothesis that the 
stick rare would spend a larger fraction of time in the vicinity of zero than would a noisy but linearly responding 
system and that, therefore, the probability density of the stick rate amplitude would be significantly non-normal and 
would show a larger peak in the vicinity of zero amplitude than a Gaussian density function. Time histories from 
the three Calspan subjects were subjected to this type of analysis. 

Computation of time-history statistics proceeded as follows: 
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b) Stick (model) 









1. A stick-rate time history was constructed by computing, at each simulation sample interval, the 
first-difference of the stick time history divided by the simulation interval. 

2. The stick-rate time history was smoothed with a rectangular window of 0.16 seconds. This smoothing 
window was increased from the 0.08 seconds used previously for plotting to minimize the effects of 
amplitude quantization on the recorded stick signal. 

3. The smoothed signal was adjusted for the estimated mean and was normalized with respect to the 
estimated standard deviation to produce a normalized "z" variate. To minimize the effects of potential 
"outliers", the median was used as the estimated mean, and the standard deviation was estimated from 
the first and third quartile boundaries. 

4. The amplitude distribution function was computed for the z variate at intervals of 0.2 for z values 
between -2.0 and 2.0. 

5. A Kolgomorov-Smimov (K-S) test was performed to determine the significance of the difference 
between the computed and theoretical normal distribution functions. 

6. The probability density function was estimated from the distribution function as follows: 

PDENS(Z) = CORK (Z) * (PDIST (Z+ , 2) -PDIST (Z- . 2) ) /0 . 4 


where PDIST is the cumulative distribution computed at Z intervals of 0.2 between -2.0 and 2.0. and 
CORR(Z) is the correction factor, computed as the ratio of the probability density of the normal curve 
divided by the density estimated from the normal distribution function according to the rule shown 
above. Because of "end effects", the probability density function was computed for z values between 
-1.8 and +1.8. 

7. The rms difference between the computed probability density and the theoretically normal probability 
density was computed to provide a quantitative measure the deviation of the computed probability 
density from normality. 


The above computations were performed on three time time histories, one for each of the test pilots. To 
provide a basis for comparison, identical analysis was performed on time histories obtained from the simulation 
model using the experimental forcing functions tracked by Pilots A and B. 

Figure 3-15 shows that the simulated stick-rate time histories are nearly Gaussian. This subjective conclusion 
is confirmed by the first two rows of Table 3-3, which show no statistically significant deviation (i.e., alpha level > 
0.05) from normality for the curve in Figure 3-15a and a marginally significant difference for the curve of Figure 
3- 15b. Thus, if the test pilots are linear but noisy responders as assumed in the OCM, we would expect the 
experimental stick-rate time histories to be nearly Gaussian. 

Figure 3-16 shows that probability densities computed from the experimental results were less Gaussian than 
those computed from the model results. Furthermore, the deviation from normality was substantially less for the 
best-performing pilot (Pilot A) than for the other two pilots; this observation is confirmed by the rms deviations 
shown in the bottom three rows of Table 3-3. 
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Table 3-3: Amplitude Statistics for Stick Rate 


Pilot 

Alpha Level 

RMS 

A (simulated) 

— 

0.017 

B (simulated) 

0.05 

0.022 

A (experimental) 

0.05 

0.028 

B (experimental) 

0.01 

0.058 

C (experimental) 

0.01 

0.054 


Alpha significance level determined from K-s test. 

RMS indicates deviation of the probability density from the normal 
curve . 


Figures 3-16a and 3-16c show trends consistent with a move-and-wait tendency in that the probability density 
is more peaked than the normal curve. This trend is particularly pronounced for Pilot C. The interpretation of the 
results for Pilot B is less clear the curve suggests that this pilot tended to let his control input slowly drift during the 
intervals between significant control movements. 

Because of the limited database explored, conclusions drawn from the foregoing analysis must be considered 
tentative at this stage. The results suggest the following trends: 

1. Better-perfoiming subjects, in terms of rms error minimization, tend to adopt a response strategy that 
is more nearly linear than subjects who perform less well. 

2. Nonlinearities present in the operator’s response strategy include move-and-wait episodes during 
which the operator maintains a relatively constant control force. 

Additional study is needed to determine the extent to which these tentative conclusions can be generalized. 
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Figure 3-15: Probability Densities for Simulated Stick Rate 
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a) Pilot A 


b) Pilot B 



Figure 3-16: Probability Densities for Experimental Stick Rate 
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4. Discussion 


To the extent that tracking error was analyzed by BBN, the trends predicted by pre-experiment model analysis 
were largely confirmed by the experimental study and replicated by post -experiment model analysis. Specifically, 

• Addition of 180 msec delay to the flight-control system caused a modest increase (around 22%) in rms 
tracking error. 

• Delay had a larger effect on performance with the simulated "F-16" than with the simulated "C-141". 

• Error scores were, on the average, larger in the ground-based simulations than in the in-flight 
simulations, but ground/flight differences were relatively small (about 10% on the average). 

• Performance trends were similar for the pitch and roll axes. 

• The model, "calibrated" for the reference conditions in pitch and roll, replicated the important 
performance trends, but tended to somewhat underestimate the quantitative effects of delay and 
simulator mode. 

The reader should note that all model results shown in this report were obtained with but two sets of model 
parameters: one set adjusted to jointly match roll-axis performance for the "F-16" and "C-141" in the single-axis, 
ground-based, zero-delay, target-tracking task, and one set adjusted to match pitch-axis performance. 

The major discrepancy between experimental and model results concerned the differences between single-task 
and combined-task performance, where the predicted differences were substantially greater than those observed in 
most cases. As suggested earlier, we suspect this discrepancy was due largely to an error in modeling strategy. 
Recall that the "single-axis" tasks required the pilot to control the airplane in both pitch and roll in response to an 
external target input in only one axis. We modeled this situation by selecting observation noise parameters 
appropriate to true single-axis tracking (i.e., aircraft motion allowed in only one axis), and then accommodating the 
combined-axis task by doubling the observation noise/signal ratio to reflect an even split of attention between the 
two axes. With the advantage of hindsight, a better strategy would seem to be to assume that the pilot must share 
attention in a near-equal manner between the two axes whenever the two axes must be controlled, even if there is an 
explicit input in only one axis, as the cross-coupling induced by the pilot and/or the vehicle dynamics will 
effectively provide inputs in both axes. 

It seems clear that selecting observation noise/signal ratios appropriate to combined-axis tracking for the 
so-called "single-axis" tasks would have allowed us to match the experimental results with "residual nose" levels 
closer to what would be associated with reasonable indifference thresholds. (Contract resources did not permit a 
repeat of the model analysis to test this hypothesis.) 
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On the basis of the results to date, we tentatively conclude that the OCM, properly calibrated, has the potential 
for interpolating the results of manned simulation studies for the type of steady-state flight tasks explored in the 
NT-33A experimental study. Although we have found the OCM to forecast performance trends with independent 
parameters selected entirely on the basis of previous results, we expect model application to be most reliable when 
the model is applied in conjunction with a manned simulation study, with parameters selected to match key 
reference conditions. Calibration against the specific data base will allow consideration factors specific to the 
simulation study, such as (1) the performance goals, motivation levels, and general piloting techniques of the 
specific test pilots; (2) the nature and amount of task-specific training received by the test pilots, and (3) other 
factors that account for the difference between laboratory tracking studies using over-trained college or high-school 
students and simulation studies using actual pilots performing representative flight tasks under constraints often 
avoided in the laboratory. 


Tentative rules for calibrating the model are: 

• Calibrate the model against the extreme conditions explored in the simulation study (e.g., widest- and 
narrowest-hand vehicle dynamics, smallest and largest delay common to all vehicles, etc.). 

• Adopt attention-related parameters appropriate to the number of axes controlled -- not to the number of 
axes containing external inputs. 

• Where control-gains are not optimized for aggressive operations, test for the need of a specific penalty 
on control force to model the pilot’s reluctance to generate large average control forces. 

• Select residual noise parameters to reflect, at a minimum, perceptual resolution limitations and the 
pilot’s lack of concern about reducing errors below some acceptable minimum. 

• In general, follow the model-matching procedure described in Section 2.2 of this report. 


The following areas of further study are suggested to develop and validate an integrated experimental and 

analytical method for determining simulator delay requirements: 

1. Validate the Interpolative Capabilities of the Model. If the OCM is to be the analytic component of an 
experimental/analytic technique for exploring delay requirements, it must be shown capable of reliably 
interpolating between the conditions for which it is calibrated. The recent NT-33A simulation study 
has provided a data base suitable for a first test of model interpolation, as it contains data from two 
additional sets of aircraft dynamics (the "C-17" and "C-21" data), plus three additional time delays for 
each of these configurations. In addition, extrapolation from the target-tracking to the gust-disturbance 
tasks remains to be accomplished. This task, then, would not require additional manned simulations. 

2. Extend OCM to Handling Qualities Predictions. Although this report has dealt entirely with objective 
performance metrics, the NT-33A data base also includes the traditional Cooper-Harper handling 
qualities metrics for each experimental condition. To the extent that pilot opinion will be considered 
an important indicator of simulator effectiveness, the utility of the OCM will be enhanced if it can 
interpolate handling qualities as well as performance data. We therefore suggest that an effort be made 
to explore a consistent method of model application that will mimic the NT-33A handling qualities 
data. This effort would build upon previous model applications in this area (Levison and Rickard, 

1981). 
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3. Validate Experimental/Analytic Methodology. This task provides a demonstration of the combined 
experimental and analytic methods for exploring time delay requirements. We suggest that the 
extrapolative capabilities of the method be tested via model analysis and manned simulation studies of 
tasks other than the attitude-tracking tasks explored in the recent Calspan program. Future studies 
should include tasks that require path as well as attitude regulation: landing approach, in-flight 
refueling, and terrain-following are potential candidates. 

4. Explore Criteria for Temporal Fidelity Requirements. The discussion so far has dealt with 
measurement and prediction of the influence of delay on objective and subjective performance, but it 
has not directly addressed the central issue of how much simulator delay (more generally, temporal 
distortion) can be tolerated in a given simulator application. This study area would most likely involve 
a significant manned simulation component, particularly if training transfer were to be explored; 
innovative modeling approaches might also be required. One possible approach to this task would be 
to develop a backward chaining of requirements, where we proceed from requirements in terms of the 
simulator application (e.g., the tolerable increments in training time), to requirements in terms of 
objective performance measures and pilot ratings, to requirements in terms of allowable delays. 
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Appendix A 


This Appendix contains population means and standard errors for tracking error and stick force SD scores for 
the conditions explored in the BBN analysis. 
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Table A-l: Experimental Roll-Axis SD Scores 

Average of 3 subjects, 2 replications/subject 



Roll Error 

Stick 

Force 

Stick 

Rate 

Vehicle Sim. Mode Delay 

Mean 

S.E. 

Mean 

S.E. 

Mean 

S.E. 

a) 1-axis Target Input 

"F-16" Ground 0 

8.23 

0.90 

1.53 

0.16 

5.14 

1.06 

180 

10.3 

1.16 

1.80 

0.16 

5.98 

0.98 

Flight 0 

7.40 

1.24 

1.30 

0.16 

5.76 

1.33 

180 

8.39 

1.03 

1.59 

0.21 

6.22 

1.83 

"C-141" Ground 0 

4.89 

0.53 

2.63 

0.29 

18.2 

1.35 

180 

6.03 

0.73 

3.22 

0.68 

13.0 

2.43 

Flight 0 

4.97 

0.98 

1.89 

0.29 

8.86 

2.54 

180 

5.66 

0.95 

2.43 

0.18 

13.3 

1.35 

b) 2-axis Target Input 

"F-16" Ground 0 

8.56 

1.36 

1.45 

0.16 

4.39 

1.08 

180 

9.76 

1.10 

1.77 

0.20 

6.14 

1.58 

"C-141" Ground 0 

5.41 

0.54 

2.46 

0.22 

8.77 

1.55 

180 

6.65 

0.78 

2.85 

0.28 

11.4 

3.17 

c) 2-axis Disturbance Input 

"F-16" Ground 0 

4.12 

0.58 

0.61 

0.15 

3.22 

1.15 

180 

4.49 

0.78 

0.81 

0.24 

4.62 

2.16 

"C-141" Ground 0 

5.35 

0.54 

2.83 

0.41 

13.0 

3.03 

180 

6.44 

0.72 

3.26 

0.54 

15.2 

5.89 
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Table A-2: Experimental Pitch-Axis SD Scores 

Average of 3 subjects, 2 replications / 3 ub ject . 


Pitch Error Stick Force Stick Rate 


Vehicle Sim. Mode Delay Mean 


a) 1-axis Target Input 


"F-16" 

Ground 

0 

0.352 



180 

0.451 


Flight 

0 

0.290 



180 

0.383 

"C-141" 

Ground 

0 

0.295 



180 

0.356 


Flight 

0 

0.284 



180 

0.352 


b) 2-axis Target 

Input 


"F-16" Ground 

0 

0.366 


180 

0.498 

"C-141" Ground 

0 

0.304 


180 

0.425 


c) 2-axis Disturbance Input 


F-16" 

Ground 

0 

0.221 



180 

0.267 

C-141" 

Ground 

0 

0.325 



180 

0.401 


S.E. 

Mean 

S.E. 

Mean 

S.E. 

0.046 

1.08 

0.05 

4.07 

0.42 

0.050 

1.22 

0.04 

5.82 

1.06 

0.039 

1.01 

0.18 

3.96 

0.90 

0.047 

1.16 

0.16 

4.90 

0.75 

0.043 

3.60 

0.14 

14 . 8 

2.89 

0.039 

4.33 

0.22 

20.4 

6.97 

0.048 

3.71 

1.19 

14.9 

3.75 

0.044 

4.38 

0.81 

31.0 

7.95 


0.036 

0.033 

0.037 

0.047 

1.34 

1.57 

4.67 

5.46 

0.07 

0.13 

0.35 

0.48 

6.92 

10.1 

20.7 

27.8 

0.61 

3.65 

3.94 

10.1 

0.011 

0.019 

0.635 

0.702 

0.049 

0.168 

6.45 

7.43 

1.09 

3.22 

0.109 

0.123 

3.58 

4.42 

0.96 

1.07 

24.9 

30.0 

12.1 

13.2 
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Table A-3: Model-Generated Roll-Axis SD Scores 


Vehicle Sim. Mode Delay 


a) 1-axis Target Input 


"F-16" 

Ground 

0 

180 


Flight 

0 

180 

"C-141" 

Ground 

0 

180 


Flight 

0 

180 

b) 2-axis Target 

Input 

"F-16" 

Ground 

0 

180 

"C-141" 

Ground 

0 

180 


Roll Stick Stick 

Error Force Rate 


8.26 

1.47 

4.95 

9.63 

1.57 

5.48 

7.81 

1.36 

4.50 

9.04 

1.42 

4.89 

5.13 

2.43 

8.45 

5.87 

2.63 

9.56 

4.75 

2.19 

7.48 

5.37 

2.36 

8.29 


10.6 1.51 5.18 

7.01 2.59 9.17 


c) 2-axis Disturbance Input 


F-16" 

Ground 

0 

180 

C-141" 

Ground 

0 

180 
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Table A-4: Model Predicted Pitch-Axis SD Scores 


Vehicle Sim. Mode Delay 


a) 1-axis Target Input 


"F-16" 

Ground 

0 

180 


Flight 

0 

180 

"C-141" 

Ground 

0 

180 


Flight 

0 

180 

b) 2-axis Target 

Input 

"F-16" 

Ground 

0 

180 

"C-141" 

Ground 

0 

180 


Pitch Stick Stick 

Error Force Rate 


0.353 

1.01 

2.70 

0.431 

1.04 

3.05 

0.339 

0.974 

2.53 

0.411 

0.985 

2.82 

0.283 

3.51 

9.27 

0.332 

3.63 

1.02 

0.276 

3.36 

8.69 

0.322 

3.43 

9.48 


0.452 1.01 2.81 

0.357 3.54 9.67 


c) 2-axis Disturbance Input 

"F-16" Ground 0 

180 

"C-141" Ground 0 

180 
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